For my first YOUmoz blog post I’d like to be a bit confrontational. Not in a nasty way, just because I’ve been searching for the answer to this question for a while. Rand posted something of interest in his 12 Easy Mistakes that Plague Newcomers to the SEO Field post about wasting PageRank on pages which are blocked with robots.txt directives, something which I had been doing and something I’d been mulling over for a while.
From visual observations of the tool bar PageRank, it would appear that pages blocked with robots directives do lose all their PageRank, but then how can Google possibly assign PageRank to a page not in the index? Therefore that page is leaking PageRank? But following a discussion at WebProWorld on the matter, I read this reply from “crankydave,” who refers to links pointing at blocked pages as “dangling links.” When you think of them in that context you read this…
“Dangling links are simply links that point to any page with no outgoing links. They affect the model because it is not clear where their weight should be distributed, and there are a large number of them. Often these dangling links are simply pages that we have not downloaded yet……….Because dangling links do not affect the ranking of any other page directly, we simply remove them from the system until all the PageRanks are calculated. After all the PageRanks are calculated they can be added back in without affecting things significantly.” Extract from the original PageRank paper by Google’s founders, Sergey Brin and Lawrence Page.
So, according to their original paper, links to pages that aren’t indexed/pages you’ve blocked are classed as dangling links and in that case are taken out of the PageRank calculation, or at least have minimal effect on it?
There are still some questions though, the main one being is this how Google’s PageRank calculations still work? Or is the minimal effect that dangling links have on PageRank distribution something worth bothering about? Also, the robots directives were around before the PageRank algorithm was even conceived, so why wouldn’t Google factor them into it from the beginning?
I still don’t really know, but it would appear thus far that Rand was wrong and that blocking pages with robots directives doesn’t cause PageRank issues. What do you think?
